The Logos Translatability Index

A serious problem for users of machine translation is estimating the suitability of a particular document for MT and, thus, the translation quality. The available utilities such as style and grammar checkers, well-formedness and readability measures have not proven sufficiently useful in this regard. Researchers at Logos Corporation therefore undertook a study of the feasibility of a translatability index (Tl) based on the use of gross sentence properties such as length and other complexity factors; on gross text properties; as well as on the idiosyncrasies of the Logos system. The study was based on English and German test data drawn from a variety of sources and representing a variety of styles and subject areas. The results of this study suggest that a statistically-based special-purpose Tl achieves significant and useful correlations with the perceived quality of raw translation, and, further, that improvement in the Tl score effected by dictionary updates and pre-edits results in corresponding improvements in output quality. The concrete results of this work are two prototypes that measure and score the suitability of English and German documents for the LOGOS translation system. 1 Issues, Problems, and Assumptions 1.1 Statistical Basis and Scope of Applicability Logos Corporation has developed a Tl to automatically assess the suitability of a given input text for MT. The Tl is based on the gross statistical properties of a document rather than on parsing its sentences. This decision was suggested by the fact that there always appeared to be a rough correlation between the quality of raw MT output and certain gross properties of a text, such as sentence length, degree of syntactic complexity, discourse characteristics, etc. It could be shown that such correlation can be found in the finer levels of statistical detail, and that it is indeed sufficiently consistent. Thus, statistical evaluation of a document can provide a measure of the translatability of that document by the LOGOS system. It was not expected, however, that the proposed method for evaluating a document on the basis of sentence properties could provide sentence-specific information with any degree of reliability. The Tl applies to the corpus or document as a whole but is not useful in pinpointing problem sentences. For this, full-scale sentence parsing would be required.